This invention relates to techniques for managing master data. Master Data Management (MDM) focuses on managing and interlinking reference data, also referred to as Master Data, which is shared by different systems and used by different groups within an organization. Some examples of Master Data include data about clients and customers, products, employees, inventory, suppliers, analytics and so on.
An important consideration when creating an MDM system is how dynamic and variable data is stored in the system. For example, product information is an example of dynamic and variable data that is conceptually represented as a hierarchical document. These hierarchical documents vary by product categories, and do typically not fit inside a traditional database schema. For example, the product attributes for a laptop computer vary greatly from the product attributes on a jet airplane or a loaf of bread.
A key issue in MDM applications, regardless of how the data is stored, is to achieve a high level of performance with the dynamic and variable data, as required by the largest retailers and suppliers that are common users of MDM systems. At the same time, however, the MDM systems must also have the ability to use standard technologies for searching and for creating views of the data.
One technique for storing dynamic and variable data, such as product data, is to store the data in a vertical table format, where the database rows contains item id, attribute id, value, and so on. This technique, however, requires many database rows and much server memory to reconstruct the tree of data.
Another technique involves storing the data in an XML (Extensible Markup Language) format only. A problem with this technique is that it is often considered to be too slow in loading the data from the database. The DOM (Document Object Model) processing of XML documents also uses a large amount of memory and CPU resources. For example, a large amount of computational work is needed to update an attribute in a document, or for validation.
Yet another technique involves storing the dynamic and variable data using a proprietary binary format technology. Representing data in a binary format provides quick and easy memory access, but does neither lend itself very well to searching on the data nor to creating views of the data. It would be desirable to have improved mechanisms for storing and retrieving dynamic and variable data.